Image processing method, carrier medium carrying image processing program, image processing apparatus, and imaging apparatus

ABSTRACT

A proposition is to provide an image processing method, a carrier medium carrying an image processing program, an image processing apparatus, and an imaging apparatus capable of attaining line reproducibility and magnification distortion reduction in a well-balanced manner in an image. The image processing method is an image processing method for performing predetermined geometric transformation processing (h(θ)) on an image to be processed, in which the predetermined geometric transformation processing (h(θ)) includes geometric transformation processing for magnification distortion reduction (h=α tan (θ/2)̂(κp)) that reduces discrepancy between circumferential magnification and radial magnification of the image to be processed, and at least one parameter (θp) that determines the strength or content of the predetermined geometric transformation processing (h=α tan (θ/2)̂(κp)) is set according to the structure of the image to be processed.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a Continuation Application of InternationalApplication No. PCT/JP2007/000393, filed Apr. 11, 2007, designating theU.S., in which the International Application claims a priority date ofMay 6, 2006, based on prior filed Japanese Patent Application No.2006-127757, the entire contents of which are incorporated herein byreference.

BACKGROUND

1. Field

The present application relates to an image processing method, a carriermedium carrying an image processing program, an image processingapparatus, and an imaging apparatus for performing geometrictransformation processing on a shot image by an optical system.

2. Description of the Related Art

Shooting lenses of general cameras are designed so that an image that issimilar to a subject plane assumed is formed on an imaging plane.Therefore, when attention is paid to a subject existing at a localposition in a shot image, the magnification in the radial direction(i.e., the direction from the optical axis center toward the imageperiphery) is larger than that in the circumferential direction (i.e.,the direction around the optical axis center). In this specification, inthe following, the discrepancy between the radial magnification and thecircumferential magnification of a subject existing at a local positionin an image will be called “magnification distortion.”

Unlike the distortion aberration, the magnification aberration occurseven if a shooting lens is free of aberrations. And the magnificationaberration corresponds to differences in impressions between a subjectas observed by the naked eyes and its shot image through a shootinglens. For example, when an observer gazes at a subject that is notlocated at the center when observing a relatively wide field with thenaked eyes, the axes of sighting of the eyeballs of the observer areinclined without the observer's realizing it. On the other hand, theoptical axis is fixed when the same field is shot through a shootinglens. Therefore, in a shot image through the shooting lens, a subjectthat is located at a peripheral position looks thicker in the radialdirection. This is magnification distortion and is more remarkable asthe position goes outward in a wide-angle image.

The geometric transformation characteristic of a projection algorithmthat is free of distortion aberration, that is, a projection algorithmthat is high in line reproducibility, is given by the followingInequality (A), where h is the image height and θ is the subject angle.This geometric transformation characteristic causes magnificationdistortion though it is high in line reproducibility.

h(θ)∝ tan θ  (A)

On the other hand, among projection algorithms of fish-eye lenses etc.is a stereographic projection algorithm that does not causemagnification distortion. The geometric transformation characteristic ofthe stereographic projection algorithm is given by the followingInequality, where h is the image height and θ is the subject angle. Thisgeometric transformation characteristic is low in line reproducibilitythough it does not cause magnification distortion.

h(θ)∝ tan(θ/2)  (B)

In view of the above, a geometric characteristic that lies midwaybetween Inequalities (A) and (B) is proposed in Patent Document 1(Japanese Unexamined Patent Application Publication No. 2005-110207).This in-between geometric characteristic is given by the followingInequality (C). In Inequality (C), P is a predetermined value thatsatisfies a relationship 1<P<4.

h(θ)∝ tan(θ/P)  (C)

Conventionally, it has been considered that both of magnificationdistortion reduction and high reproducibility can be attained byperforming geometric transformation processing on a shot image by acamera according to this geometric transformation characteristic.

However, the geometric transformation characteristic of Inequality (C)does not always make it possible to express the entire area of an imagesatisfactorily and noticeable line distortion or magnificationdistortion may occur depending on the position in an image.

SUMMARY

A proposition is to solve the above problems and thereby provide animage processing method, a carrier medium carrying an image processingprogram, an image processing apparatus, and an imaging apparatus capableof attaining line reproducibility and magnification distortion reductionin a well-balanced manner in an image.

An image processing method according to the invention is an imageprocessing method which performs predetermined geometric transformationprocessing on an image to be processed, wherein the predeterminedgeometric transformation processing includes geometric transformationprocessing for magnification distortion reduction that reducesdiscrepancy between circumferential magnification and radialmagnification of the image to be processed, and at least one parameterthat determines strength or content of the geometric transformationprocessing for magnification distortion reduction is set according tostructure of the image to be processed.

The geometric transformation processing for magnification distortionreduction may be such as to act on a peripheral area of the image to beprocessed.

The predetermined geometric transformation processing may includegeometric transformation processing for magnification distortionreduction that acts on a peripheral area of the image to be processedand geometric transformation processing for line reproduction that actson a central area of the image to be processed.

A position of a boundary between the peripheral area and the centralarea may be set as the parameter.

It is desirable that at least one of h(θ), h′(θ) as derivatives of h(θ),and κ(θ) be continuous at θp, where θ is an arbitrary subject angle, θpis a subject angle corresponding to the boundary position, h(θ) is animage height, as a function of the subject angle θ, obtained afterexecution of the predetermined geometric transformation processing, andκ(θ) is a ratio between the circumferential magnification and the radialmagnification at the subject angle θ.

The image processing method may be such that the image to be processedhas magnification distortion that is symmetrical with respect to anoptical axis center, the geometric transformation processing formagnification distortion reduction has a geometric transformationcharacteristic that is not symmetrical with respect to the optical axiscenter, and strength q of the geometric transformation processing formagnification distortion reduction is set as the parameter.

It is desirable that the strength q be set in a range of 0<q<1.

It is desirable that the geometric transformation processing formagnification distortion reduction has a geometric transformationcharacteristic that is symmetrical with respect to a vertical line and ahorizontal line that pass through the optical axis center.

The image processing method may be such as to display images obtainedbefore and/or after execution of the predetermined geometrictransformation processing to a user and to let the user set theparameter.

The image processing method may be such as to perform line detection onthe image to be processed and set the parameter according to its result.

The geometric transformation processing for magnification distortionreduction may be geometric transformation processing which converts theimage to be processed into an image that would be obtained when the samesubject were shot by a virtual optical system whose optical axis islocated at a position that is deviated from a center of the image to beprocessed, and the optical axis position of the virtual optical systemmay be set as the parameter.

The position of the optical axis of the virtual optical system may beset at a main subject position in the image to be processed.

The image processing method may be such as to display the image to beprocessed to a user and to let the user specify the main subjectposition.

The image processing method may be such as to perform image recognitionon the image to be processed and to detect the main subject positionbased on its result.

The predetermined geometric transformation processing may includegeometric transformation processing for distortion aberration reductionthat reduces distortion aberration of the image to be processed inaddition to the geometric transformation processing for magnificationdistortion reduction, and pixel interpolation processing may beperformed at one time on the image that has been subjected to both ofthe geometric transformation processing for magnification distortionreduction and the geometric transformation processing for distortionaberration reduction.

A carrier medium carrying an image processing program according to theinvention causes a computer to execute the image processing methodaccording to any of the image processing methods according to theinvention.

An image processing apparatus according to the invention performs any ofthe image processing methods according to the invention.

An imaging apparatus according to the invention includes any of theimage processing apparatus according to the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates magnification distortion that is symmetrical withrespect to the optical axis center.

FIG. 2 are front views, as viewed along the optical axis, of a sphericalimage and a flat image.

FIG. 3 illustrates a geometric transformation characteristic accordingto a first embodiment.

FIG. 4 shows a relationship between the subject angle θ and the imageheight h₂ of the geometric transformation characteristic according tothe first embodiment.

FIG. 5 shows a relationship between the subject angle θ and themagnification distortion ratio κ of the geometric transformationcharacteristic according to the first embodiment.

FIG. 6 is an operation flowchart of a computer according to a thirdembodiment.

FIG. 7 is an operation flowchart of a computer according to a fourthembodiment.

FIG. 8A to FIG. 8C illustrate a procedure of a fifth embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS First Embodiment

A first embodiment will be described. This embodiment is an embodimentof an image processing method for performing magnification distortioncorrection on an image to be processed. In this embodiment, it isassumed that the image to be processed is a shot image through a generalshooting lens having a small distortion aberration.

In the magnification distortion correction according to the embodiment,a pixel value of a pixel having coordinates (x, y) in an image to beprocessed is moved to a pixel value of a pixel having coordinates (x₂,y₂) by geometric transformation processing that consists of thefollowing <operation 1> to <operation 4> and pixel interpolationprocessing is performed.

<Operation 1>

An image height h corresponding to coordinates (x, y) is calculatedaccording to the following equation:

[Formula 1]

h=√{square root over ((x−cx)²+(y−cy)²)}{square root over((x−cx)²+(y−cy)²)}  (1)

In Equation (1), (cx, cy) are coordinates of the optical axis center inthe image to be processed.

<Operation 2>

A subject angle θ corresponding to the image height h is calculatedaccording to the following equation:

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 2} \right\rbrack & \; \\{\theta = {\arctan \left( {\frac{d_{\max}}{f}\frac{h}{h_{\max}}} \right)}} & (2)\end{matrix}$

In Equation (2), d_(max) is a maximum image height as converted into alength in the image to be processed, f is the focal length of theshooting lens, and h_(max) is a maximum image height as converted intothe number of pixels in the image to be processed.

<Operation 3>

The subject angle θ is converted into an image height h₂ according tothe following equations:

$\begin{matrix}\left\lbrack {{Formulae}\mspace{14mu} 3} \right\rbrack & \; \\\left\{ \begin{matrix}{{{h_{2}(\theta)} = \left\{ \begin{matrix}{{C\; \frac{h_{\max}}{d_{\max}}f\; \tan \; \theta}\mspace{14mu}} & {{{if}\mspace{14mu} \theta} \leq {\theta \; p}} \\{{C_{2}\; \frac{h_{\max}}{d_{\max}}{f\left( {\tan \; \frac{\theta}{2}} \right)}^{\kappa \; p}}\;} & {{{if}\mspace{14mu} \theta} > {\theta \; p}}\end{matrix} \right.}} & \; \\{{{\kappa \; p} = \frac{1}{\cos \; \theta \; p}}} & \; \\{{C_{2} = {C\; \tan \; \theta \; p \times \left( {\tan \; \frac{\theta \; p}{2}} \right)^{{- \kappa}\; p}}}} & \;\end{matrix} \right. & (3)\end{matrix}$

In Equation (3), C is the variable magnification and is set at a propervalue of about 1 to 1.2, for example, so as to avoid excessive imagereduction. θp is a threshold value and is set according to the structureof the image to be processed. When an image to be processed has ageneral structure, it is appropriate to set θp at about 40°.

<Operation 4>

Coordinates (x₂, y₂) corresponding to the image height h₂ are calculatedaccording to the following equations:

$\begin{matrix}\left\lbrack {{Formulae}\mspace{14mu} 4} \right\rbrack & \; \\{x_{2} = {{cx} + {\left( {x - {cx}} \right) \times \frac{h_{2}}{h}}}} & \left( {4\; x} \right) \\{y_{2} = {{cy} + {\left( {y - {c\; y}} \right) \times \frac{h_{2}}{h}}}} & \left( {4\; y} \right)\end{matrix}$

Advantages of First Embodiment

First, a description will be made of magnification distortion that issymmetrical with respect to the optical axis center. An image to beprocessed of this embodiment and an image that has been subjected to themagnification distortion correction according to the embodiment can beregarded as having magnification distortion that is symmetrical withrespect to the optical axis center.

As shown in FIG. 1, assume a case of forming a subject image onto asphere image by a pinhole camera. In this case, the distance betweenpoints P and Q in the spherical image is proportional to a view anglebetween points P₀ and Q₀ that is obtained when the subject is seen fromthe pinhole. That is, the spherical image has no magnificationdistortion. However, in actual cameras, since a subject is reproduced asa flat image by a geometric characteristic that is symmetrical withrespect to the optical axis center, magnification distortion occurs thatis symmetrical with respect to the optical axis center. Therefore, themagnification distortion of the flat image can be calculated byconsidering local variations in image magnification that occur when thespherical image is projected onto the flat image.

FIG. 2 are front views, as viewed along the optical axis, of thespherical image and the flat image.

It is assumed that the feature point P in the spherical image isprojected to a feature point P₂ in the flat image, and that the featurepoint Q which is separated from the feature point P by a very smalldistance is projected to a feature point Q₂ in the flat image.

The coordinates of the feature point P are determined by an angle(subject angle) θ that is formed by the optical axis and the straightline connecting the pinhole and the feature point P (see FIG. 1) and anangle (circumferential angle) φ of the straight line around the opticalaxis (see FIG. 2). The coordinates of the feature point P are thusrepresented by (θ, φ).

Likewise, the coordinates of the feature point Q are determined by asubject angle (θ+Δθ) and a circumferential angle (φ+Δφ). The coordinatesof the feature point Q are thus represented by (θ+Δθ, φ++Δθ).

On the other hand, the coordinates of the feature point P₂ arerepresented by an image height and a circumferential angle in the flatimage. The image height of the feature point P₂ is represented by afunction h(θ) which is a function of only the subject angle θ of thefeature point P, and the circumferential angle of the feature point P₂is the same as the circumferential angle φ of the feature point P(because the geometric transformation characteristic is symmetrical withrespect to the optical axis center). The coordinates of the featurepoint P₂ are thus represented by (h(θ), φ).

Likewise, the coordinates of the feature point Q₂ are represented by animage height and a circumferential angle in the flat image. The imageheight of the feature point Q₂ is represented by a function h(θ+Δθ)which is a function of the subject angle (θ+Δθ) of the feature point Q,and the circumferential angle of the feature point Q₂ is the same as thecircumferential angle (φ+Δφ) of the feature point Q. The coordinates ofthe feature point Q₂ are thus represented by (h(θ+Δθ), φ+Δφ).

As shown in the left-hand part of FIG. 2, if the distance between thefeature points P and Q in the spherical image is approximated by astraight line, the circumferential component d1φ and the radialcomponent d1θ of the distance are given by the following equations,respectively, using the distance r from the spherical image to thepinhole (see FIG. 1).

[Formulae 5]

d1φ=(r sin θ)Δφ  (5φ)

d1θ=rΔθ  (5θ)

As shown in the right-hand part of FIG. 2, the circumferential componentd2φ and the radial component d2 h of the distance between the featurepoints P₂ and Q₂ in the flat image are given by the following equations,respectively:

$\begin{matrix}\left\lbrack {{Formulae}\mspace{20mu} 6} \right\rbrack & \; \\{{d\; 2\; \varphi} = {{h(\theta)}\; \Delta \; \varphi}} & \left( {6\; \varphi} \right) \\\begin{matrix}{{d\; 2\; h} = {{h\left( {\theta + {\Delta \; \theta}} \right)} - {h(\theta)}}} \\{= {\frac{d\; {h(\theta)}}{d\; \theta}\Delta \; \theta}}\end{matrix} & \left( {6\; h} \right)\end{matrix}$

Therefore, the circumferential magnification kφ and the radialmagnification kh in the flat image are given by the following equations,respectively:

$\begin{matrix}\left\lbrack {{Formulae}\mspace{14mu} 7} \right\rbrack & \; \\{{k\; \varphi} = {\frac{d\; 2\; \varphi}{d\; 1\; \varphi} = \frac{h(\theta)}{r\; \sin \; \theta}}} & \left( {7\; \varphi} \right) \\{{k\; h} = {\frac{d\; 2\; h}{d\; 1\; \theta} = {\frac{1}{r}\frac{d\; {h(\theta)}}{d\; \theta}}}} & \left( {7\; h} \right)\end{matrix}$

Therefore, the magnification distortion ratio κ in the flat image isgiven by the following equation:

$\begin{matrix}\left\lbrack {{Formula}\mspace{14mu} 8} \right\rbrack & \; \\{{\kappa \equiv \frac{k\; h}{k\; \varphi}} = \frac{\sin \; \theta \; \frac{d\; h(\theta)}{d\; \theta}}{h(\theta)}} & (8)\end{matrix}$

For example, in the image to be processed of the embodiment, the subjectis elongated in the radial direction in an area where the magnificationdistortion ratio κ is larger than 1. In the distortion aberrationcorrection for the image to be processed, it is necessary to make themagnification distortion ratio κ fall within an allowable range (e.g.,smaller than or equal to 1.3) in the entire area of the image to beprocessed. In particular, the magnification distortion ratio κ tends tobe large in a peripheral area of the image to be processed (i.e., in anarea where the subject angle θ is large). It is important to reduce themagnification distortion ratio κ in the peripheral area.

Next, the geometric transformation characteristic according to theembodiment will be considered with reference to FIG. 3.

In the geometric transformation characteristic according to theembodiment, a geometric transformation characteristic that causes theimage height h₂ to be proportional to tan θ, that is, a geometrictransformation characteristic that is high in line reproducibility, isemployed in an area where the subject angle θ is smaller than or equalto the threshold value θp (i.e., a central area A1 in FIG. 3).

However, geometric transformation characteristics that are high in linereproducibility generate magnification distortion. A magnificationdistortion ratio that is caused by the geometric transformationcharacteristic that is high in line reproducibility is calculated as1/(cos θ) by substituting h(θ)∝ tan θ into Equation (8). That is, thegeometric transformation characteristic that is high in linereproducibility generates larger magnification distortion as the subjectangle θ increases.

Furthermore, in the geometric transformation characteristic according tothe embodiment, a geometric transformation characteristic that causesthe image height h₂ to be proportional to tan (θ/2)̂(κp), that is, ageometric transformation characteristic that reduces the magnificationdistortion, is employed in an area where the subject angle θ is largerthan the threshold value θp (i.e., a peripheral area A2 in FIG. 3).

A magnification distortion ratio that is caused by the geometrictransformation characteristic that reduces the magnification distortionis calculated as κp=1/(cos θp) by substituting h(θ)∝ tan (θ/2)̂(κp) intoEquation (8). That is, the geometric transformation characteristic thatreduces the magnification distortion keeps the magnification distortionthe same as that at θ=θp irrespective of the subject angle θ.

FIG. 4 shows a relationship between the subject angle θ and the imageheight h₂ of the geometric transformation characteristic according tothe embodiment. FIG. 5 shows a relationship between the subject angle θand the magnification distortion ratio κ of the geometric transformationcharacteristic according to the embodiment. In FIGS. 4 and 5, thethreshold value θp is set at 40° and data of a geometric transformationcharacteristic h₂(θ)∝ tan θ are shown additionally for comparison.

As shown in FIG. 4, the content of the geometric transformationcharacteristic according to the embodiment is switched at the boundarybetween the area of θ≦θp (central area A1) and the area of θp<θ(peripheral area A2). As a result of the switching, as shown in FIG. 5,the magnification distortion ratio κ increases gradually in the area ofθ≦θp (central area A1) and is kept at a constant value κp in the area ofθp<θ (peripheral area A2). The magnification distortion ratio κ iscontinuous at θ=θp, and also in the range of θ≦θp it is kept to thevalue κp taken in the area of θp<θ (peripheral area A2).

As described above, in the magnification distortion correction accordingto the embodiment, as shown in FIG. 3, importance is attached to theline reproducibility in the central area A1 of the image to be processedand to the magnification distortion reduction in the peripheral area A2.

In the embodiment, the magnification distortion can fall within theallowable range in the entire area merely by setting the threshold valueθp so that the magnification distortion falls within the allowable rangein the peripheral area A2 by taking the structure etc. of the image tobe processed into consideration.

Incidentally, it is desirable that a default threshold value θp be setat about 40°, for example. In this case, the magnification distortionratio κ is reduced to about 1.3 or less.

In particular, if the threshold value θp is set so that themagnification distortion in the peripheral area A2 becomes the maximumvalue of the allowable range, the deterioration of the linereproducibility in the peripheral area A2 can be reduced to a minimumnecessary level and hence the line reproducibility and the magnificationdistortion reduction are balanced well in the entire area.

Furthermore, in the geometric transformation processing according to theembodiment, h(θ) which represents the relationship between the imageheight h and the subject angle θ, h′ (θ) as the derivative of h(θ), andthe magnification distortion ratio κ(θ) each are continuous at θ=θpwhere the geometric transformation characteristic is switched, thecentral area A1 and the peripheral area A2 are connected to each othersmoothly and naturally at the boundary a.

Supplements to First Embodiment

The image to be processed of the embodiment is a shot image through ageneral shooting lens having a small distortion aberration. However,images taken through other kinds of shooting lenses may be processed.

For example, a shot image through a fish-eye lens may be an image to beprocessed of the embodiment. However, in this case, it is necessary touse a geometric transformation characteristic (i.e., an equationrepresenting a relationship between the image height h and the subjectangle θ) that is specific to the fish-eye lens in place of Equation (2)in the above-described <operation 2>. Taking an image (image to beprocessed) through a fish-eye lens is suitable for monitoring camerasystems etc. because a very wide angle, natural image is obtained.

A shot image through a shooting lens having a large distortionaberration may be an image to be processed of the embodiment. However,in this case, it is necessary to acquire distortion aberrationinformation of the shooting lens in advance and to insert, between theabove-described <operation 2> and <operation 3>, a geometrictransformation operation of converting the image height h based on thedistortion aberration information (i.e., geometric transformationprocessing for distortion aberration correction).

In the magnification distortion correction according to the embodiment,the pixel interpolation processing is performed on an image that hasbeen subjected to the geometric transformation processing. The pixelinterpolation processing is processing of interpolating pixel values atinteger coordinate positions using pixel values at decimal coordinatepositions that have been subjected to the geometric transformation. Sucha technique as cubic interpolation can be applied to this processing.

Where plural kinds of geometric transformation processing are combinedto the magnification distortion correction according to the embodiment,it is desirable that the pixel interpolation processing be performed atone time after execution of all of the plural kinds of geometrictransformation processing.

In the geometric transformation processing according to the embodiment,setting the variable magnification C at 1 in Equations (3) means thatthe geometric transformation processing for the central area A1 of theimage to be processed is omitted. Depending on the structure of an imageto be processed, the geometric transformation processing need not beperformed for the central area A1. This setting makes it possible toshorten the processing time of the magnification distortion correctionby omitting useless calculations.

The image to be processed of the embodiment is a shot image through anactual shooting lens. However, images taken through lenses that are notan actual shooting lens may be processed as long as a subject isprojected by a geometric transformation processing that is symmetricalwith respect to the image center.

For example, an image that is obtained by cutting out part of a shotimage through a fish-eye lens and performing geometric transformationprocessing on a resulting image so as to eliminate distortion aberrationmay be an image to be processed of the embodiment. Since such an imagehas magnification distortion that is similar to that of a shot imagethrough a general shooting lens having a small distortion aberration,the embodiment can be applied to it. That is, the term “optical axiscenter” as used in the embodiment encompasses a virtual optical axiscenter of such an image.

The magnification distortion correction according to the embodiment canbe performed in image processing apparatus capable of capturing animage, such as a computer, an image storage unit, and a printer. Forexample, where it is performed in a computer, it is appropriate toprepare a program for performing the magnification distortion correctionaccording to the embodiment in advance and install it in the computer.Alternatively, similar processing may be performed on a mobile apparatushaving an imaging function, such as a digital camera.

The magnification distortion correction according to the embodimentrequires such information as a focal length f of a shooting lens and amaximum image height length d_(max). Where the magnification distortioncorrection according to the embodiment is performed by an imageprocessing apparatus such as a computer, a storage unit, or a printer,it is appropriate for a user to input those pieces of information to theimage processing apparatus.

Alternatively, where adjunct information is attached to an image file ofan image to be processed (where such a file format as EXIF is employed),an image processing apparatus may automatically recognize a focal lengthf of a shooting lens, a maximum image height length d_(max), etc. basedon the adjunct information and a separately provided database.

Second Embodiment

A second embodiment will be described. This embodiment is also anembodiment of an image processing method in which an image to beprocessed is an image having magnification distortion that issymmetrical with respect to the optical axis center and magnificationdistortion correction is performed on it. Differences from the firstembodiment will be described mainly.

The differences reside in the details of the geometric transformationprocessing. In the geometric transformation processing according to theembodiment, a pixel value of a pixel having coordinates (x, y) in animage to be processed is moved to a pixel value of a pixel havingcoordinates (x₂, y₂) that are given by the following Equations (9):

$\begin{matrix}\left\lbrack {{Formulae}\mspace{14mu} 9} \right\rbrack & \; \\\left\{ \begin{matrix}{{{f(x)} = {\int_{0}^{x}{\frac{1}{\kappa (h)}\ {h}}}}} \\{{{g(x)} = {{\left( {1 - q} \right) \times x} + {q \times {f(x)}}}}} \\{{x_{2} = {{cx} + {C \times {g\left( {x - {cx}} \right)}}}}} \\{{y_{2} = {{cy} + {C \times {g\left( {y - {cy}} \right)}}}}}\end{matrix} \right. & (9)\end{matrix}$

In Equations (9), κ(h) is the magnification distortion ratio at theimage height h in the image to be processed and (cx, cy) are coordinatesof the optical axis center in the image to be processed. And C is thevariable magnification and is set at a proper value of about 1 to 1.2,for example, so as to avoid excessive image reduction.

Parameter q represents the strength of the geometric transformationprocessing and is set according to the structure of the image to beprocessed. The strength q is set in a range of 0≦q≦1 and desirably in arange of 0≦q≦0.7.

Advantages of Second Embodiment

Since the geometric transformation characteristic according to theembodiment is symmetrical with respect to the vertical line and thehorizontal line that pass through the optical axis center, high linereproducibility is obtained in the vertical direction and the horizontaldirection of an image to be processed. Furthermore, this geometrictransformation characteristic reduces the magnification distortion ratioκ at individual positions in the vertical direction and the horizontaldirection with respect to the optical axis center.

With such a geometric transformation characteristic, no straight linesin the vertical direction or the horizontal direction are distorted inthe entire area of an image to be processed and the magnificationdistortion of a subject located over, under, to the right of, or to theleft of the optical axis center is reduced.

However, the geometric transformation processing according to theembodiment is low in line reproducibility in oblique directions.Therefore, if its strength q is high, oblique lines in an image to beprocessed are distorted unnaturally.

In view of this, in the embodiment, the strength q of the geometrictransformation processing is set at a proper value that is smaller than1 taking the structure etc. of an image to be processed intoconsideration. As a result, the line reproducibility and themagnification distortion reduction can be well balanced in the entirearea of the image to be processed.

Incidentally, it is desirable that the default value of the strength qof the geometric transformation processing according to the embodimentbe set at about 0.4, for example. To prevent unnatural distortion ofoblique lines, the strength q needs to be set at least smaller than orequal to 0.7. However, in general, line structures of an image to beprocessed are in many cases extend in the vertical direction or thehorizontal direction. Therefore, in many images to be processed, it ispossible to set the strength q at values that are close to 1.

Supplements to Second Embodiment

In the geometric transformation processing characteristic according tothe embodiment, the strength q of geometric transformation processing inthe vertical direction (y direction) and that in the horizontaldirection (x direction) have the same value. However, they may be set atdifferent values.

The magnification distortion correction according to the embodiment canbe performed in image processing apparatus capable of capturing animage, such as a computer, an image storage unit, and a printer. Forexample, where it is performed in a computer, it is appropriate toprepare a program for performing the magnification distortion correctionaccording to the embodiment in advance and install it in the computer.Alternatively, similar processing may be performed on a mobile apparatushaving an imaging function, such as a digital camera.

Magnification distortion correction may be performed by combining thegeometric transformation processing according to this embodiment andthat according to the first embodiment. This magnification distortioncorrection consists of an operation of performing the geometrictransformation processing according to the first embodiment on an imageto be processed, an operation of performing the geometric transformationprocessing according to this embodiment, and an operation of performingpixel interpolation processing. It is desirable that the pixelinterpolation processing be performed at one time after execution of allthe kinds of geometric transformation processing.

Third Embodiment

A third embodiment will be described. This embodiment is an embodimentof an image processing program that is suitable for a case that themagnification distortion correction according to the first embodiment isperformed by a computer. This image processing program causes a computerto operate in the following manner. It is assumed that a monitor as anoutput device and a mouse, a keyboard, etc. as input devices areconnected to the computer and serve for interfacing with a user.

FIG. 6 is an operation flowchart of the computer according to theembodiment. As shown in FIG. 6, the computer reads an image to beprocessed (operation S11) and sends the image to be processed to themonitor to display it (operation S12). A user can recognize thestructure of the image to be processed on the monitor. The computerrecognizes a focal length f of a shooting lens, a maximum image heightlength d_(max), etc. by referring to adjunct information of an imagefile of the image to be processed and a separately prepared database.

The computer causes a user to set the geometric transformation parameterof the magnification distortion correction (operation S13). Thegeometric transformation parameter is the threshold value θp in theabove-described Equations (3). It is appropriate for the user to set thethreshold value θp small if he or she judges that importance should beattached to the magnification distortion reduction rather than the linereproducibility, and to set the threshold value θp large if he or shejudges that importance should be attached to the line reproducibilityrather than the magnification distortion reduction.

When the user has set the geometric transformation parameter (in thisembodiment, the threshold value θp) (operation S13: yes), the computerperforms, in a simplified manner, the magnification distortioncorrection according to the first embodiment on the image to beprocessed (operation S14) and displays a corrected image (simplifiedimage) on the monitor (operation S15). The simplified magnificationdistortion correction means magnification distortion correction on asize-reduced version of the image to be processed, magnificationdistortion correction with simplified pixel interpolation processing, orthe like.

The user can judge from the simplified image on the monitor whether thegeometric transformation parameter (in this embodiment, the thresholdvalue θp) set by himself or herself is proper. If the user judges thatthe geometric transformation parameter is improper and sets a geometrictransformation parameter (in this embodiment, a threshold value θp)(operation S16: no; operation S13: yes), operations S14 and S15 areexecuted again. The user can adjust the geometric transformationparameter (in this embodiment, the threshold value θp) any number oftimes until he or she is satisfied with a simplified image on themonitor (operations S13, S14, and S15).

Then, if the user is satisfied with a simplified image on the monitorand inputs an execution instruction to perform magnification distortioncorrection to the computer (operation S16: yes), the computer performsthe magnification distortion correction according to the firstembodiment on the image to be processed in a detailed manner (operationS17) using the currently set geometric transformation parameter (in thisembodiment, the threshold value θp) and displays a corrected image(detailed image) on the monitor (Operation S18). The detailedmagnification distortion correction means magnification distortioncorrection on a non-size-reduced version of the image to be processed.

Then, if the user inputs an image storage instruction to the computer(operation S19), the computer stores the detailed image in a nonvolatilememory such as a hard disk in a proper file format (operation S20).

Advantages of Third Embodiment

The image processing program according to the embodiment displays animage to be processed and a simplified image on the monitor. Therefore,a user can cause the computer to perform desired magnificationdistortion correction while checking the structure of the image to beprocessed and the effects of the magnification distortion correction.For example, the user can attach importance to the line reproducibilityby setting the threshold value θp large if the image to be processed isan image of a building and can attach importance to the magnificationdistortion reduction by setting the threshold value θp small if theimage to be processed is a group photograph.

In the image processing program according to the embodiment, themagnification distortion correction of operation S14 is the simplifiedmagnification distortion correction. Therefore, operations S13, S14, andS15 are executed at high speed. Therefore, a user can check the effectsof the magnification distortion correction in real time while adjustingthe geometric transformation parameter (in this embodiment, thethreshold value θp).

Supplements to Third Embodiment

In this embodiment, the magnification distortion correction according tothe first embodiment is employed. Alternatively, the magnificationdistortion correction according to the second embodiment may beemployed. In this case, the geometric transformation parameter to be setby a user is the strength q in Equations (9).

In this embodiment, the magnification distortion correction according tothe first embodiment is employed. Alternatively, magnificationdistortion correction may be employed in which the geometrictransformation processing according to the first embodiment and thataccording to the second embodiment are combined together (described inthe “supplements to first embodiment” section. In this case, thegeometric transformation parameters to be set by a user are both of thethreshold value θp in Equations (3) and the strength q in Equations (9).

Although the embodiment uses the computer, similar processing may beperformed on an image processing apparatus capable of capturing an imagesuch as an image storage unit or a printer instead of the computer. As afurther alternative, similar processing may be performed on a portableapparatus having an imaging function, such as a digital camera.

Fourth Embodiment

A fourth embodiment will be described. This embodiment is an embodimentof an image processing program for automating the magnificationdistortion correction using a computer. This embodiment is directed to acase of employing magnification distortion correction in which thegeometric transformation processing according to the first embodimentand that according to the second embodiment are combined together. It isassumed that a monitor as an output device and a mouse, a keyboard, etc.as input devices are connected to the computer and serve for interfacingwith a user.

FIG. 7 is an operation flowchart of the computer according to theembodiment. As shown in FIG. 7, the computer reads an image to beprocessed (operation S11). At this operation, the computer recognizes afocal length f of a shooting lens, a maximum image height lengthd_(max), and distortion aberration information by referring to adjunctinformation of an image file of the image to be processed and aseparately prepared database. Among these pieces of information, thedistortion aberration information is recognized based on information oflens positions at the time of shooting that is attached to the imagefile and the database. However, if the distortion aberration informationis attached to the image file, it is not necessary to refer to thedatabase.

Then, the computer generates a simplified image for line detection byperforming size reduction transformation processing, geometrictransformation processing for distortion aberration correction, andpixel interpolation processing in this order on the image to beprocessed (operation S20), and extracts line information from thesimplified image (operation S21). The line information indicates whatand how many straight lines exist in the image to be processed. A lineinformation extraction method of operation 521 will be described below.

Line Information Extraction Method

First, edges of the simplified image are detected and a binary edgeimage is generated which consist of edges whose values are larger thanor equal to a predetermined threshold value. A Hough image is generatedby analyzing the edge image. In generating a Hough image, first,Equation (10) is employed as a line model. The origin of x-y coordinatesis the optical axis center.

x cos θ+y sin θ=ρ  (10)

Then, a (θ, ρ) combination that satisfies the following Equation (11) isfound for edge coordinates (x_(i), y_(i)) in the edge image.

x _(i) cos θ+y _(i) sin θ=ρ  (11)

Then, the thus found (θ, p) are mapped onto the O-p coordinates. Therange of θ is 0≦θ≦180°. The mapping means adding “1” to the pixel valueof a pixel corresponding to (θ, ρ) on the θ-ρ coordinates. The Houghimage is an image that is formed finally on the θ-ρ coordinates byrepeating the mapping for all pairs of edge coordinates (x_(i), y_(i))of the edge image.

Then, a normalized Hough image is acquired by normalizing the Houghimage and pixels having local maximum pixel values in the normalizedHough image are determined.

Coordinates (θ, ρ) of each of these pixels indicate a type of straightlines that are included in the image to be processed by a predeterminedamount or more. The pixel value H of each of those pixels indicates anamount of each straight line. Therefore, it is appropriate to employ (θ,ρ, H) of each of those straight lines as line information of the imageto be processed. (The description of the line information extractionmethod ends here).

Then, the computer calculates an amount A_(H1) of straight lines thatwill be distorted by the geometric transformation processing accordingto the first embodiment (operation S22) and calculates an amount A_(H2)of straight lines that will be distorted by the geometric transformationprocessing according to the second embodiment (operation S24) based onthe line information.

Straight lines that will be distorted by the geometric transformationprocessing according to the first embodiment are straight lines that donot pass through the vicinity of the optical axis center and whose pvalues are larger than a predetermined value (because p represents thedistance from the straight line to the optical axis center as seen fromEquation (10)). The sum of H values of these straight lines is theamount A_(H1) of straight lines that will be distorted by the geometrictransformation processing according to the first embodiment.

Straight lines that will be distorted by the geometric transformationprocessing according to the second embodiment are straight lines each ofwhich is neither a vertical line nor a horizontal line and whose θvalues are different from 0°, 90°, or 180° by a predetermined value ormore (because θ is the angle formed by the straight line and the y axisas seen from Equation (10)). The sum of H values of these straight linesis the amount A_(H2) of straight lines that will be distorted by thegeometric transformation processing according to the second embodiment.

Then, the computer sets the geometric transformation parameter θp at avalue that is suitable for the amount A_(H1) of straight lines(operation S23). A function θp(A_(H1)) is a monotonically increasingfunction in a broad sense; θp increases with A_(H1). A manufacturer ofthe program may properly set the function θp(A_(H1)) based onexperiments or simulations conducted in advance. It is appropriate toset the minimum value of θp at about 30°.

The computer sets the geometric transformation parameter q at a valuethat is suitable for the amount A_(H2) of straight lines (operationS24). A function q(A_(H2)) is a monotonically decreasing function in abroad sense; q decreases with A_(H2). A manufacturer of the program mayproperly set the function q(A_(H2)) based on experiments or simulationsconducted in advance. It is appropriate to set the maximum value of q atabout 0.7.

Then, the computer performs geometric transformation processing fordistortion aberration correction using the distortion aberrationinformation, the geometric transformation processing according to thefirst embodiment using the geometric transformation parameter θp, thegeometric transformation processing according to the second embodimentusing the geometric transformation parameter q, and pixel interpolationprocessing in this order on the image to be processed (detailed image),and completes the magnification distortion correction (operation S26).

Then, if necessary, the computer stores an image obtained by themagnification distortion correction in a nonvolatile memory such as ahard disk in a proper file format (operations S19 and S20).

Advantages of Fourth Embodiment

The image processing program according to the fourth embodiment sets thegeometric transformation parameters automatically according to thestructure of an image to be processed. This saves a user time and laborand thus provides high convenience. Furthermore, the followingadvantages are provided:

(1) Since the geometric transformation processing for distortionaberration correction is performed on an image to be processed beforeextraction of line information from the image to be processed, theaccuracy of the extraction can be increased.

(2) Since the size reduction transformation processing is performed onan image to be processed before extraction of line information from theimage to be processed, the speed of the extraction processing can beincreased.

(3) Since Hough transform is used for the extraction of lineinformation, the accuracy of the extraction can be increased.

(4) Since the strength of geometric transformation processing of such akind that line structures are distorted is set at a proper low level,the line reproducibility can be kept at a high level.

(5) Since the strength of geometric transformation processing of such akind that line structures are not distorted is set at a proper highlevel, both of high line reproducibility and a high level ofmagnification distortion reduction can be attained.

Supplements to Fourth Embodiment

In this embodiment, the geometric transformation processing fordistortion aberration correction may be omitted in the case wheredistortion aberration information cannot be acquired. Certain degrees ofadvantages can be obtained even in such a case.

In this embodiment, the magnification distortion correction is employedin which the geometric transformation processing according to the firstembodiment and that according to the second embodiment are combinedtogether. Alternatively, one of the magnification distortion correctionaccording to the first embodiment and that according to the secondembodiment may be employed.

Although the embodiment uses the computer, similar processing may beperformed on an image processing apparatus capable of capturing an imagesuch as an image storage unit or a printer instead of the computer. As afurther alternative, similar processing may be performed on a portableapparatus having an imaging function, such as a digital camera.

Fifth Embodiment

A fifth embodiment will be described. This embodiment is an embodimentof an image processing method for performing magnification distortioncorrection on a shot image through a shooting lens. This embodiment willbe described with an assumption that the magnification distortioncorrection is performed by a computer. It is also assumed that a monitoras an output device and a mouse, a keyboard, etc. as input devices areconnected to the computer and serve for interfacing with a user.

The computer executes the following <operation 1> to <operation 4>according to a preinstalled image processing program.

<Operation 1>

As shown in FIG. 8A, an image to be processed is displayed on themonitor. In FIG. 8A, S1 and S2 denote subjects and C1 denotes acrosshairs cursor. The crosshairs cursor C1 points the subject S1 thatis located in a central area of the image to be processed.

<Operation 2>

As shown in FIG. 8B, a user is caused to specify a main subjectposition. FIG. 8B shows a state that the user has moved the crosshairscursor C1 to specify the subject S2 that is located in a peripheral areaof the image to be processed.

<Operation 3>

A virtual optical system is assumed whose optical axis is located at thespecified position. For example, the geometric transformationcharacteristic of the virtual optical system is such as to provide arelationship h(θ)∝ tan θ between the subject angle θ and the imageheight h.

The image to be processed is converted into an image that would beobtained when the same subjects were shot by the virtual optical systemby performing, on the image to be processed, predetermined geometrictransformation processing in which the above geometric transformationcharacteristic is taken into consideration. Furthermore, pixelinterpolation processing is performed on a converted image.

The predetermined geometric transformation processing is a combinationof geometric transformation processing whose characteristic is inverseto a geometric transformation characteristic specific to the shootinglens, rotational transformation for converting the optical axisdirection of the shooting lens so that it coincides with the opticalaxis direction of the virtual optical system, and geometrictransformation processing whose geometric transformation characteristicis specific to the virtual optical system. Incidentally, where theshooting lens is a general one having a small distortion aberration, theinverse characteristic is given by the above-described Equation (2).

Incidentally, a technique similar to the above geometric transformationprocessing is disclosed in Japanese Unexamined Patent ApplicationPublication No. H06-121318. However, in this embodiment, the geometrictransformation is performed on the entire area of the image to beprocessed instead of its partial area. In the embodiment, a convertedimage is trimmed into a rectangular shape if necessary.

<Operation 4>

A processed image is displayed on the monitor. FIG. 8C shows a displayedimage. As shown in FIG. 8C, the magnification distortion of the subjectS2 in the peripheral area that was designated by the user is reduced. Onthe other hand, the magnification distortion of the subject S1 in thecentral area is increased.

Advantage of Fifth Embodiment

This embodiment makes it possible to attain magnification distortionreduction and line reproducibility only for a main subject designated bya user in an image to be processed.

Supplements to Fifth Embodiment

In this embodiment, a main subject position is specified by a user.Alternatively, a main subject position may be detected automatically insuch a manner that the computer performs image recognition processingsuch as face detection on an image to be processed.

It is desirable that the trimming of the embodiment be such as to leaveas wide an area as possible (at least a half area).

Where the image to be processed of the embodiment is a shot imagethrough a shooting lens having a large distortion aberration, it isappropriate to acquire distortion aberration information of the shootinglens in advance and incorporate geometric transformation processing (fordistortion aberration correction) for converting the image height hbased on the distortion aberration information into the above-describedpredetermined geometric transformation processing.

Although the embodiment uses the computer, similar processing may beperformed on an image processing apparatus capable of capturing an imagesuch as an image storage unit or a printer instead of the computer. As afurther alternative, similar processing may be performed on a portableapparatus having an imaging function, such as a digital camera.

The many features and advantages of the embodiments are apparent fromthe detailed specification and, thus, it is intended by the appendedclaims to cover all such features and advantages of the embodiments thatfall within the true spirit and scope thereof.

Further, since numerous modifications and changes will readily occur tothose skilled in the art, it is not desired to limit the inventiveembodiments to the exact construction and operation illustrated anddescribed, and accordingly all suitable modifications and equivalentsmay be resorted to, falling within the scope thereof.

1. An image processing method which performs predetermined geometric transformation processing on an image to be processed, wherein: said predetermined geometric transformation processing includes geometric transformation processing for magnification distortion reduction that reduces discrepancy between circumferential magnification and radial magnification of said image to be processed; and at least one parameter that determines one of strength and content of said geometric transformation processing for magnification distortion reduction is set according to structure of said image to be processed.
 2. The image processing method according to claim 1, wherein said geometric transformation processing for magnification distortion reduction acts on a peripheral area of said image to be processed.
 3. The image processing method according to claim 2, wherein said predetermined geometric transformation processing includes geometric transformation processing for magnification distortion reduction that acts on a peripheral area of said image to be processed and geometric transformation processing for line reproduction that acts on a central area of said image to be processed.
 4. The image processing method according to claim 3, wherein a position of a boundary between said peripheral area and said central area is set as said parameter.
 5. The image processing method according to claim 4, wherein at least one of h(θ), h′(θ) as derivatives of h(θ), and κ(θ) is continuous at θp, when θ is an arbitrary subject angle, θp is a subject angle corresponding to said boundary position, h(θ) is an image height, as a function of said subject angle θ, obtained after execution of said predetermined geometric transformation processing, and κ(θ) is a ratio between the circumferential magnification and the radial magnification at said subject angle θ.
 6. The image processing method according to claim 1, wherein said image to be processed has magnification distortion that is symmetrical with respect to an optical axis center, said geometric transformation processing for magnification distortion reduction has a geometric transformation characteristic that is not symmetrical with respect to the optical axis center, and strength q of said geometric transformation processing for magnification distortion reduction is set as said parameter.
 7. The image processing method according to claim 6, wherein said strength q is set in a range of 0<q<1.
 8. The image processing method according to claim 6, wherein said geometric transformation processing for magnification distortion reduction has a geometric transformation characteristic that is symmetrical with respect to a vertical line and a horizontal line that pass through said optical axis center.
 9. The image processing method according to claim 1, comprises displaying images obtained at least one of before and after execution of said predetermined geometric transformation processing to a user and letting the user set said parameter.
 10. The image processing method according to claim 1, comprises performing line detection on said image to be processed and setting said parameter according to its result.
 11. The image processing method according to claim 1, wherein said geometric transformation processing for magnification distortion reduction is geometric transformation processing for converting said image to be processed into an image that may be obtained when the same subject is shot by a virtual optical system whose optical axis is located at a position that is deviated from a center of said image to be processed, and said optical axis position of said virtual optical system is set as said parameter.
 12. The image processing method according to claim 11, wherein said position of the optical axis of said virtual optical system is set at a main subject position in said image to be processed.
 13. The image processing method according to claim 12, comprises displaying said image to be processed to a user to let the user specify said main subject position.
 14. The image processing method according to claim 12, comprises performing image recognition on said image to be processed and detecting said main subject position based on its result.
 15. The image processing method according to claim 1, wherein: said predetermined geometric transformation processing includes geometric transformation processing for distortion aberration reduction for reducing distortion aberration of said image to be processed in addition to said geometric transformation processing for magnification distortion reduction; and pixel interpolation processing is performed at one time on the image that has been subjected to both of said geometric transformation processing for magnification distortion reduction and said geometric transformation processing for distortion aberration reduction.
 16. A carrier medium carrying an image processing program causing a computer to execute the image processing method according to claim
 1. 17. An image processing apparatus performing the image processing method according to claim
 1. 18. An imaging apparatus comprising the image processing apparatus according to claim
 17. 